Stochastic Tree Search: Where to Put the Randomness?

In this short note, I argue against two commonlyheld biases. The first is that stochastic search is applicable only to improvement search over complete solutions. On the contrary, many problems have effective greedy heuristics for constructing solutions, making a tree-structured search space more appropriate. The second is that stochastic tree search algorithms should explore the same space of decisions as systematic methods. Constructing search trees in the traditional manner, by choosing the default variable at the parent and valuing it differently at each child, makes sense for efficient complete search, but is not necessarily the best choice for incomplete methods. In an empirical study using the combinatorial optimization problem of number partitioning, I show that the opposite approach, choosing a different variable at each child and giving it the default value, can be a good choice for incomplete stochastic algorithms. 1 Stochastic Tree Search A large number of papers have appeared in recent years (including at AI conferences such as IJCAI and AAAI) devoted to stochastic improvement search for optimization problems, in which an algorithm attempts to improve a complete but potentially suboptimal solution. Many of these ‘local search’ methods, such as tabu search or simulated annealing, are completely general and use as their only source of problemspecific information the ability to evaluate the objective function on a complete solution. Others, such as WalkSAT, take advantage of heuristic guidance in the form of a function that identifies variables that might be profitably changed. Improvement methods are often contrasted with complete search methods, which use techniques such as branch-and-bound or dynamic backtracking [Ginsberg, 1993] to systematically extend an empty solution in all possible ways, implicitly traversing a tree containing all possible solutions. When run to completion, such methods guarantee an optimal solution. But it ∗Division of Engineering and Applied Sciences, 33 Oxford Street, Cambridge, MA 02138 USA. www.eecs.harvard.edu/ ̃ruml. is easy to over-generalize and assume that stochastic search is applicable only to improvement search. In fact, the call for papers for this workshop exhibits this tendency. It summarizes the field of interest as “stochastic local search algorithms, as well as randomised systematic search methods,” leaving the impression that stochastic search applies only to local improvement and that tree-structured search spaces are best left to systematic methods (perhaps suitably randomized). Such a bias is ill-founded for two reasons. First, there are many problems for which heuristic guidance is available when the problem is considered as a tree. And second, as I will explain, trees provide an convenient avenue for attacking one of the main research questions in stochastic search: how to represent information learned during the search. For many problems, effective heuristic guidance is available when viewing the problem as a tree search. This is evidenced by the fact that greedy constructive heuristics exist for many problems. Such methods can be viewed as incomplete tree searches that visit only a single leaf without backtracking [Korf, 1995]. Expanding a node in the tree corresponds to extending a partial solution, and the leaves of the tree correspond to complete solutions which can be evaluated according to the objective function. The greedy algorithm provides a way of choosing which component of the problem should be set next (with analogy to CSPs, call this variable choice) and what it should be set to (call this value choice). Systematic algorithms such as depth-first search and limited discrepancy search [Harvey and Ginsberg, 1995] can exploit this heuristic knowledge, and the fact that such algorithms can perform well on certain problems despite their fixed and non-adaptive search order indicates that this heuristic guidance is powerful indeed. Stochastic approaches are often inherently incomplete. But for large problems in which the complete tree cannot be enumerated, there is no reason to think that stochastic tree search algorithms could not do better than existing systematic methods. Several stochastic tree searches have already been proposed. Simple unguided random probing was found effective on scheduling benchmarks by Crawford and Baker [1994]. Bresina’s [1996] Heuristic-Biased Stochastic Sampling (HBSS) makes repeated probes into a tree, weighting its choice of child according to the value-ordering function. The Greedy Randomized Adaptive Search Procedure (GRASP) approach of Feo and Resende [1995] is actually a combination of heuristically biased probing with improvement search on the resulting leaf (see also Marchiori and Steenbeek [1998]). Juillé and Pollack [1998] use random tree probing as a value choice heuristic during a beam search into a tree, and Abramson proposed similar methods for game trees [1991]. Stochastic tree search provides an avenue for addressing one of the important active research questions in stochastic search: how to explicitly represent information learned during the search so that it can be used to guide future actions? (The collection of papers edited by Boyan, Buntine, and Jagota [2000] surveys current work in this area.) In local search, it is difficult to formulate a representation that can capture the past history of objective values because the search space is so unstructured. Tabu search methods implicitly represent regions of the search space which should not be explored. Boyan’s [2000] STAGE system requires usersupplied subroutines that calculate features of solutions. It then uses them to generalize about good regions from which to start improvement search. Boese et al.’s [1994] Adaptive Multi-Start (AMS) uses solutions themselves as stand-ins to mark good regions of the search space, but this representation again requires a user-supplied combining function to implement generalization. Baluja’s [1997] Population-Based Incremental Learning (PBIL) works only with binary problems, and hence can use a probability vector to represent learned information about which variables should be 0 or 1. But in tree-based stochastic search, the tree itself can provide the geometry of the search space. As the work on discrepancy search shows [Harvey and Ginsberg, 1995; Korf, 1996; Walsh, 1997], one can usefully generalize across levels of the tree. I have been working on adaptive probing algorithms [Ruml, 2001] that learn at what depths of the tree one can trust the given child-ordering function. A stochastic framework is employed that chooses the heuristicallypreferred child at each level according to the estimated probability that it is in fact the better choice. This varies according to the number of probes that have been performed and the data obtained. The ability to generalize about the childordering heuristic across the breadth of the tree provides a concise representation of the learned probing bias. (Adaptive probing may also help provide a principled grounding for algorithms such as HBSS, GRASP, and the Ant Colony Optimization work of [Dorigo and Gambardella, 1997], in which ‘pheremone’ accumulates to represent the combined information gathered by multiple search trials.) The Squeaky Wheel Optimization (SWO) method of Joslin and Clements [1998] is another adaptive tree search method. In SWO, a greedy algorithm is used to construct solutions given an order in which to consider the variables. A variable choice function (similar to those used in improvement search) identifies variables that are poorly set in the resulting complete solution. Those variables are moved earlier in the ordering, usually resulting in their being handled better by the greedy heuristic. One can think of this technique as adapting the given variable choice heuristic. Unlike many improvement search methods, stochastic tree search is not necessarily ‘local’—it might visit very different solutions on consecutive iterations. But like improvement methods, this flexibility comes at the cost of incompleteness, and one must be alert to the possibility of expending redundant effort in the same part of the search space (a generalization of the ‘local minima’ of improvement search). Given the relative abundance of heuristic information in a tree-structured search space, and the way that the clear geometry of the tree provides a convenient form for representing learned information during search, stochastic tree search seems ripe for research attention. 2 Alternative Tree-Structured Search Spaces One of the most fundamental questions in tree search is how to trade-off the information provided by the variable-choice heuristic and the value-choice heuristic. More simply put, what is the best way to structure the search? Complete systematic search methods typically trust the variable-choice heuristic completely. Children of a node vary in the value they assign to that variable; they can be ordered according to the value-choice heuristic. This results in a depth n tree with the branching factor depending on the number of values per variable. (Typically, the number of values is smaller than the number of variables.) This represents the smallest tree that completely enumerates the possibilities (size b). But efficiently enumerating all solutions is already out of the question for many real-world problems. In the context of incomplete search, it may in fact be more productive to acknowledge that the variable choice function could be fallible too. For some problems, it may be useful to consider an alternative tree representation in which one searches not over the value assignments but over the variable choices. Each child of a node would represent the choice of a different variable, which would be assigned the value chosen by the value choice heuristic. The children could be ordered according to the variable choice heuristic. From a conventional point of view, searching this tree is ludicrous. It is much larger than the conventional one, as it has a branching factor of O(n), as well as depth n, for a size of n!, and it does not necessarily even include all possible solutions. But the solutions that it does contain may be of a higher quality, depending on the accuracy of the value choice heuristic. It is important to recognize that the choice of search space representation is not pre-ordained—the bias towards conventional representation may be inappropriate for incomplete methods. 2.1 Empirical Investigation To demonstrate the relevance of this question, I have performed some experiments with the combinatorial optimization problem of number partitioning. The objective in a number partitioning problem is to divide a given set of numbers into two disjoint groups such that the difference between the sums of the two groups is as small as possible. It was used by Johnson et al. to evaluate simulated annealing [1991], Korf to evaluate his improvement to limited discrepancy search (ILDS) [1996], and Walsh to evaluate depth-bounded discrepancy search (DDS) [1997]. When the numbers are chosen uniformly over an interval, the difficulty of the problem depends on the relation between the number of digits in the numbers and the number of numbers. With few digits and L og 10 (D if fe re nc e) -2